Carnegie  Mellon 

Software  Engineering  Institute 


Pittsburgh,  PA  15213-3890 


Advanced  Risk  Analysis  for 
High-Performing  Organizations 


Christopher  Alberts 
Audrey  Dorofee 


Sponsored  by  the  U.S.  Department  of  Defense 
©  2006  by  Carnegie  Mellon  University 


page  1 


Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1.  REPORT  DATE 

2QQg  2.  REPORT  TYPE 

3.  DATES  COVERED 

00-00-2006  to  00-00-2006 

4.  TITLE  AND  SUBTITLE 

Advanced  Risk  Analysis  for  High-Performing  Organizations 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Carnegie  Mellon, Software  Engineering  Institute, 4500  Fifth 

Avenue, Pittsburgh, PA, 15213-2612 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS (ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF:  17.  LIMITATION  OF 

ARSTRATT 

1 8 .  NUMBER  1 9a.  NAME  OF 

OF  PAGES  RESPONSIBLE  PERSON 

a.  REPORT  b.  ABSTRACT  c.  THIS  PAGE  Same  aS 

unclassified  unclassified  unclassified  Report  (SAR) 

37 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


Carnegie  Mellon 

Software  Engineering  Institute 


Changing  Operational  Environment 


From 

Centralized  management 
control  of  processes 

Dedicated,  stand-alone 
technologies 

Permanent  enterprise,  defined 
by  organizational  chart 

One  team,  one  mission 

Compartmentalized  view  of  risk 
(e.g.,  project,  security) 


To 

Distributed  management 
control  of  processes 

Interoperable,  networked 
technologies 

Virtual  enterprise,  defined  by 
mission 

Many  teams,  one  mission 
Integrated  view  of  risk 
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Changing  Risk  Profiles 

Changes  in  operational  environments  are  driving  the  need 
for  advanced  risk  analysis  techniques. 

•  The  operational  environment  is  becoming  more 
complex  (e.g.,  distributed  processes). 

•  New  types  of  risks  have  emerged  from  this  complexity. 

-  inherited  risk 

-  new  sources  of  risk  (e.g.,  cyber-security  risks) 

-  risk  from  combinatorial  effects 

-  risk  from  cascading  consequences 

-  risk  from  emergent  threats 
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The  Need  for  Advanced  Techniques 

High-performing  organizations  are  able  to  manage 
traditional  risks. 

Risks  arising  from  operational  complexity  are  often  subtle 
in  nature,  but  bring  the  potential  for  catastrophic 
consequences. 

High-performing  organizations  have  the  basic  skills 
needed  to  manage  these  new  types  of  risk,  but  sufficient 
techniques  are  not  readily  available. 
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Key  Requirements 

High  performers  need  advanced  risk  management  techniques 
that  enable  them  to 

•  assume  an  integrated  view  of  risk  (one  view  that  includes 
process,  technology,  security,  and  interoperability  risks) 

•  address  the  interrelated  nature  of  risk  (combinatorial  effects 
and  cascading  consequences) 

•  understand  the  amount  of  risk  that  is  inherited  from  partners 
and  collaborators 

•  characterize  the  risk  arising  from  the  emergent  properties  of  a 
distributed  process 
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What  Is  Risk? 

The  possibility  of  suffering  harm  or  loss 

Risk  requires  the  following  conditions: 

•  loss 

•  uncertainty 

•  choice 
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Nature  of  Risk 

Speculative  (dynamic)  -  a  risk  that  has  profit  and  loss 
associated  with  it 

Hazard  (static)  -  a  risk  that  only  has  loss  associated  with  it 

Profit 
Status  Quo 
Loss 

Speculative  Hazard 
Risk  Risk 
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Operational  Risk1 

The  risk  of  loss  resulting  from  inadequate  or  failed  internal 
processes,  people  and  systems,  or  from  external  events 


1.  Bank  for  International  Settlements  (BIS).  International  Convergence  of  Capital  Measurement 
and  Capital  Standards:  A  Revised  Framework.  BIS,  2004.  http://www.bis.org/publ/bcbs107.pdf. 
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Sources  of  Risk  During  Operations 


Mission 


Environment 


Design 


Categories 
of  Threat 


Event 


Activity 


A  broad  range  of  threats  must  be  considered  when 
analyzing  the  potential  for  mission  success. 
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Mission 


A  mission  threat  is  a  fundamental  flaw,  or  weaknesses, 
the  purpose  and  scope  of  a  work  process. 
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Process  Design 


A  design  threat  is  an  inherent  weakness  in  the  layout  of  a 
work  process. 
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Activity  Management 


An  activity  threat  is  a  flaw,  or  weaknesses,  arising  from  the 
manner  in  which  activities  are  managed  and  performed. 
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Operational  Environment 


Operational  Environment 


An  environment  threat  is  an  inherent  constraint,  weakness, 
or  flaw  in  the  overarching  operational  environment  in  which  a 
process  is  conducted. 
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Event  Management 


An  event  threat  is  a  set  of  circumstances  triggered  by  an 
unpredictable  occurrence  that  introduces  unexpected 
change  into  a  process. 


©  2006  by  Carnegie  Mellon  University 


page  14 


Carnegie  Mellon 

Software  Engineering  Institute 

Mission  Risk 

The  possibility  that  a  mission  might  not  be  successfully 
achieved 
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Mission  Assurance 

Establishing  a  reasonable  degree  of  confidence  in  mission 
success 

Mission  assurance  is  achieved  by  ensuring  that  risk  to  the 
mission  (i.e.,  mission  risk)  is  within  tolerance. 

A  key  aspect  of  mission  assurance  is  its  dual  focus  on 
outcome  and  execution. 
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Mission  Assurance  Strategy 
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What  is  MAAP? 

MAAP  is  a  protocol,  or  heuristic,  for  determining  the 
mission  assurance  of  an  operational  process  or  system. 
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Key  Characteristics  of  MAAP 

Applies  an  engineering  approach  to  risk  analysis 

Designed  for  highly  complex  environments  (multi¬ 
organization,  system  of  systems) 

Provides  an  in-depth  analysis  of  processes,  relationships, 
and  dependencies 

Characterizes  the  risk  of  mission  failures 

•  process  performance  risk 

•  security  risk 

•  operational  environment  risk 
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Structured  Analysis  of  Performance 

MAAP  analyzes  process  performance  in  multiple 
operational  states 

•  normal,  or  expected,  operational  conditions 

•  unusual  circumstances,  or  occurrences,  triggered  by 
external  events 
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Analyzing  Multiple  States 


State  1 :  Expected 

Operational 

Conditions 


Event  1 


State  2:  When  Stressed 
by  Event  1 
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operational  con 

ditions 

Event  2 


State  3:  When  Stressed 
by  Event  2 
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Risk  resulting 
from  event  1 
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Risk  resulting 
from  event  2 


Risk  to  the 
mission 
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Risk  Causal  Chain 


vulnerabilities  and  controls  operational  circumstances 


Risk  to  the 
mission 


Mission  risk 
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Bringing  Risk  within  Tolerance 


▲ 


Severe 


High 


Medium 


Low 


There  is  a  significant  gap 
y  between  actual  risk  exposure 
and  management’s  goal. 


J 


Risk  tolerance 


Minimal 


Current  value  of  mission 
risk  exposure 


Time 

Management’s  goal  for 
mission  risk  exposure 
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Key  Risk  Drivers 


Risk  during 
expected 


Risk  to  the 
mission 


A  critical  path  analysis  identifies  the  key  risk  drivers. 
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Protocol  Fundamentals  - 1 

•  Determine  mission  objectives. 

•  Characterize  all  operations  conducted  in  pursuit  of  the 
mission. 

•  Define  risk  evaluation  criteria  in  relation  to  the  mission 
objectives. 

•  Identify  potential  failure  modes. 

•  Perform  a  root  cause  analysis  for  each  failure  mode. 
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Protocol  Fundamentals  -  2 

•  Develop  a  risk  profile  of  the  mission. 

•  Ensure  that  mission  risk  is  within  tolerance. 
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A  Common  Basis  for  Analysis 


a- 


■ 
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Operational 

Security 

Analysis 
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MAAP  Pilot 


Analyzed  an  incident  management  process  in  a  large 
government  organization 

Analyzed  risk  to  the  mission  under  normal  conditions 

•  quality  of  response 

•  timeliness  of  response 

•  customer  satisfaction 

Examined  risk  to  the  mission  under  unusual 
circumstances 

•  two  major  incidents  occur  at  the  same  time 

•  cyber  security  attack  renders  ticketing  system 
unavailable  for  an  extended  period  of  time 
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Example:  Process  Workflow 
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Example:  Complex  Risks 


IRC  partnership  determines  who  fills  what  position. 
The  best  person  is  not  always  selected . 
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Example:  Mission  Risk 
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Example:  Mission  Assurance  Goal 

Management’s  goal  is  to  build  a  “world-class”  incident 
management  capability. 

This  goal  translates  to  very  high  mission  assurance  (i.e., 
very  low  risk  to  the  mission). 
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Example:  Gap  in  Performance 


▲ 


Severe 


High 


Medium 


Low 


A 


There  is  a  significant  gap 
between  actual  performance 
and  management’s  goal. 


Minimal 


Risk  tolerance 


Current  value  of  mission 
risk  exposure 


Time 

Management’s  goal  for 
mission  risk  exposure 
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Example:  Mitigation  Strategy 

•  Simplify  the  mission. 

-  Determine  which  incident  management  services 
are  essential. 

-  Develop  a  plan  for  growing  the  incident 
management  capability  over  time. 

•  Redesign  the  process  based  on  the  revised  mission. 

•  Develop  and  test  contingency  plans. 
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Conclusions 

Many  types  of  risk  prevalent  in  today’s  operational  environments 
(e.g.,  event  risks,  inherited  risk)  are  not  readily  identified  using 
traditional  risk  analysis  techniques. 

High-performing  organizations  have  the  basic  skills  needed  to 
identify  and  manage  these  new  types  of  risk,  but  lack  sufficient 
techniques. 

Average  or  poor  performers  will  not  have  the  skills  needed  to 
identify  and  manage  new  types  of  risk  (and  probably  have  bigger, 
more  obvious  risks  to  deal  with). 

MAAP  is  one  technique  that  high  performers  can  use  to  identify 
and  mitigate  the  risks  arising  from  operational  complexity. 
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Additional  Research  and  Development 

Develop  a  technique  for  quickly  estimating  mission  risk 
exposure. 

•  First  pilot  will  focus  on  mission  assurance  in  incident 
management. 

•  Second  pilot  will  focus  on  mission  assurance  in  system 
development. 

Refine  and  document  MAAP  based  on  pilot  experience. 
Pilot  MAAP  in  another  domain. 
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Telephone 
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WWW 
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U.S.  mail 
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